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ABSTRACT 

An overview of the Air Force's Research and 
Development Program for the Improvement of Maintenance Efficiency is 
provided. First described are the steps found in any detailed task 
analysis, a process which results in the complete specification of 
each task involved in an overall maintenance effort. The factors 
influencing maintenance effectiveness are listed and the structure of 
maintenance jobs outlined. Job Performance Aids (JPA) , which are 
step-by-step guidance procedures designed to reduce maintenance 
personnel training needs and to improve maintenance performance, are 
discussed, along with the more general concept of Job Oriented 
Training and the specific problem of the Job Task Performance Tests 
required for reliable, valid measurement. The results of two 
experiments are also summarized; these indicate that inexperienced 
personnel using JPA perform maintenance tasks better than experienced 
personnel using traditional technical manuals. This supports the 
conclusion that the use of JPA improves both maintenance personnel 
training and the performance of maintenance tasks. (LB) 
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in the past 20 years different agencies of the Army, Navy and 
Air Force have addressed this problem (12). But before address- 
ing measurement I would like to tell you something about Job 
Performance Aids and its integration with Job Oriented Training. 

JOB PERFORMANCE AIDS 

The JPA technology as developed by the Air Force Human 
Resources Laboratory are not as some people think, just a book 
of step-by-step job procedures with supporting pictures. Although » 
that is what the casual observer sees * Wtiat is not seen and what 
gives JPAs their effectiveness is the careful systematic control 
of their content. JPAs are the optimum end product of a comprehensive 
developmental process. This process starts with a complete task 
analysis. What is unique about the task analysis is its systematic, 
consistent and integrated consideration of many factors affecting 
the performance of each maintenance task identified by the task 
analysis. I want to emphasize the word each * Some of these factors 
include the type of equipment, its maintenance concept, the levels 
of maintenance to vjhich tasks are assigned, the type of test equip- 
ment, the aptitude of the user, and the "head^book" trade off. By 
"head-book trede off" Is meant what the user is expected to bring 
to the task that is stored in the man as against what is stored in 
the aid. A little later this morning Dr. Jack Folley will tell you 
something about the development of these structured task analysis 
procedures and their resulting sub-products which, after development, 
serve as criteria for the content of the aids (2, 15). 

J 

Factors Affecting Maintenance Effectiveness 

This development of a workable procedure for the systematic 
consideration of such factors did not just happen. In 1961 and 



1962 Dr. John A. Modrlck of the then Training Research Division of the 
Behavioral Sciences Laboratory of the Aerospace Medical Research Labor- 
atories and Dr. John D. Folley, Jr. of the American Institutes for 
Research started by giving systematic consideration to many factors 
concerned with the effectiveness of maintenance. For this consideration 
an equation such as the following was used: 

M^^^ - f (E, TE, HT, D, P) 

The effectiveness of maintenance (M^^f) is a function of several 
gross factors such as the characteristics of the equipment to be 
maintained (E) ; the test equipment (TE) and hand tools (HT) to be 
used during maintenance; the technical Information and/or instruc- 
tions provided (D) ; and the relevant skills of the people assigned 
to perform the maintenance (?) . (Each of these functions can be 
expanded into a list of sub-functions. In the interest of time I 
have delegated a fuller expansion of this formula to appendix A«} 

Traditionally, the variable in this equation, which has been 
expected to compensate for the weaknesses or shortcomings of the 
other variables, has been the "peopl«>" variable. Because of the 
high turnover of maintenance personnel In the military setting, 
on the average, "people" have had the shortest life cycle of any 
of the other gross variables. As a result of high turnover during 
the life cycle of most equipments or systems, the training of 
people to perform maintenance has been extremely expensive over 
the life cycle of most equipment or hardware systems* As a result 
important questions to be answered in 1962 were, and in 1972 still 
are: "What can be done to reduce costly training?", '*What can be 
done to Improve the performance of the maintenance man?". 
Dr. Modrick hypothesized thit Improved job information and instruc- 
tions were likely candidates. 



The Structure of the Maintenance Job 



After a review of the literature (17) and some preliminary 
development and tryouts of aids (13, 18), a Maintenance Task 
Simulator of a Bomb Navigational System was built and an indepth 
study (8, 9, 10) was made of checkout procedures. It was found 
that step-by-step guidance greatly reduced the need for training 
and improved the performance of checkout tasks (7) • Thie work 
was expanded to between stage troubleshooting and later to within 
stage troubleshooting. In all of this early work, remove and replace 
actions and use of test equipment were assigned to training. 

A three dimensional functional structure or working model 
of the Air Force maintenance job such as shown on this slide is 
a convenient way of presenting what gradually emerged. The 
various types of maintenance activities Just mentioned are repre- 
sented by one axis of the model. Since mechanical equipment and 
electronic equipment usually require a different variety of 
maintenance actions « they are represented by £uiother axis. 

The third axis represents the three levels or categories of 
maintenance now found in the military services. Organizational 
maintenance Is the first level. It is usually aimed at checking 
out a whole machine subsystem and correcting any Identified faults 
as quickly as possible. Flight line maintenance falls in this category. 
A system is checked out. If it does not work, the line replaceable 
unit (LRU) or '*black box" causing the malfunction is Identified 
and replaced. The major component is then taken to the field 
shop (intermediate maintenance) where it is again checked out and 
the faults, authorized for correction , are corrected. The corrective 
actions* authorized at the intermediate level, vary greatly from 
system to system depending on the maintenance concept of each system. 
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On some systems, the maintenance man will troubleshoot th^ black 
box to the piece part level* In more modern equipment, he will 
identify a replaceable module made up of many piece parts* Some 
modules are thrown away, other sent to the depot for repair. 
Any line replaceable units which the field shop are unable, or 
unauthorized to repair are sent to the depot for overhaul. 

Organizational and intermediate level organizations are manned 
primarily by enlisted technicians whose average length of service 
is rather short (slightly more than 4 years in the Air Force), 
Depots are manned largely by civilian personnel with a much higher 
level of experience and longer retention time. Using this mo<iel it 
has been possible to develop job performance aids that considered 
training, aptitude, and Cixperieuce of personnel variables as vrell 
as equipment and test equipment variables . 

History and Results of Job Perf ormance Aid Programs 

7 As stated earlier the AFHRL work in 1962-63 addressed che. kout 

8 procedures of electronic equipment. In the 1963-68 time frame this 
work expanded to between and within stage troubleshooting; som» align 
and adjust procedures; and remove and replace activities. In i.he ^ 

9 1966-68 time frame a controlled experiment (11) was conducted that 
indicated that (with only 12 hours of specific training on how to 
use test equipment, how tp use "hand tools"), both High and Medium 
aptitude high school subjects could perform checkout procedures, 
troubleshooting, and remove and replace activities in an equally 
acceptable manner as experienced, high aptitude, experienced, 
personnel with many months of training. The high school subjects 
used job performance aids; the experienced technicians used Technical 
Order materials. This was an encouraging finding. As a result in 
the 1968-1970 Fully Proceduralized Aids were being developed for 

a doppler radar, the AN/APN-1A7, and its computer the AN/ASN-35. 
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These aids covered both organ i /at i :>nal and intermediate niaintenance 
but only for maintenance tasks on electronic equipment . 

From 19b4 to 1969 a concurrent effort called PIMO was being 
conducted under the management of the Ballistic Systems Division 
(now SAMSO) of the Air Force Systems Command (PIHO is an acronym for 
Presenti^tion of Information for Maintenance and Operation). Although 
managed by BSD, the Behavioral Sciences Laboratory was actively 
represented on the steering committee - first by Dr. Kirk Johnson, 
and later by Mr. Jerry Chubb and myself. (Jerry Chubb Is one of our 
discussants this morning. The principal investigator for the PIMO 
project was Dr. Kay Inaba of Serendipity, Inc. He Is now president 
of XYZYX Information Corporation.) This effort addressed itself to 
the Organizational Maintenance Tasks of the C-lAl Aircraft. As you 
can see from the slide, fully proceduralized job performance aids 
called Job guide manuals were prepared for the straight line tasks- 
checkout; align and adjust; and remove *nd replace tasks. These 
aids covered both mechanical and electronic tasks. But for trouble- 
shooting an older type of Technical Data, called SIMMs (Symbolic 
Integrated Maintenance Manuals), was used. The effectiveness of 
the fully proceduralized aids were compared with technical orders in 
a controlled experimept at Charleston AFB, S. Car. The results of 
this work are reported in an eight volume Technical Report (19, 20, 
21. 23, 25, 26, 27 & 29). Volumes I and II (19, 23) of this technical 
report indicate that experienced aircraft mechanics, apprentice aircraft 
mechanics and personnel from unrelated career fields were able to 
perform error free maintenance when using the job guide manuals. 
^ The experienced personnel made errors when performing with tecbnii:al 
orders, and the apprentices and other personnel were not able to 
perform adequately with Technical Orders. 

Vletnamlzation and the Development of Specifications 

The promising results of this experiment and the AFHRL experiment 
discussed earlier led to applying Job Performance Aids to selected 



systems for Vietnamization. In Dec 1969 the first Development 
Directive was issued for Aids for the flight line maintenance on 
the UH-IH. This was followed by the C-123 and the CH-47. The 
Air Force experience with buying Job Performance Aids for the UH-IH 
indicated the need of a more precise specification which would 
insure the adequate and consistent consideration of all job tasks 
and the development of adequate and consistent instructions for 
accomplishing each task. This was done by requiring the development 
of a structured task analysis and a number of sub-products. These 
sub-products must be used by the Job Performance Aids manager 
purchasing the aids to guarantee the quality and quantity of the 
content of the aids (2). In addition "How to" handbooks were prepared 
for Job Performance Aids Developers and Managers (3, 4). The specifi- 
cation and handbookp^n my opinion^ were critical requirements - until 
they were developed we did not have a technology. These guidance 
documents have since been modified for American use as a three volume 
AFHRL Technical Report (15, 16, 22), 

The documents have been used to purchase aids for 7 additional 
systems under contracts to Westinghouse and STC for the Aeronautical 
Systems Division of AFSC. Another important application has been the 
development of Job Performance Aids for all three levels of main- 
tenance for an electronic system called Seek Point* This particular 
extension of the technology is being developed under contract by 
Applied Science Associates for the Electronic Systems Division of AFSC, 

As an exploratory development project, AFHRL has developed JPAs 
for Organizational and Intermediate maintenance of a J-85 jet engine- 
expanding our experience to include mechanical tasks for the inter- 
mediate level of maintenance. Summarizing our coverage we have now 
developed fully proceduralized aids for all types of maintenance 
except mechanical tasks at depot level. 



As indicated earlier we now have Americanized draft specifica- 
tions and guidance for the development of aids for both mechanical 
and electronic maintenance tasks at the Organizational level. These 
documents have been published as AFHRL-Technical Report-yi-SS 
(13, 16, 22), I have a few of these with me for those interested. 
In addition AFHRL has published a companion document that extends the 
application of the TR-71-23 to include not only organizational but also 
intermediate maintenance for electronic subsystems (6)« AFHHL has an ^ 
effort in progress which will update these documents and expand them to 
include Intermediate maintenance. These documents should be published 
early in 1973* 

JOB PERFORMANCE AIDS AND JOB ORIENTED TRAINING 

All of the above job performance aids efforts have had as a 
common objective the improvement of the performance of maintenance. 
The AFHRL has had added the objective of improved training. As a 
part of our advanced development program, we have developed matching 
Job oriented training for jobs modified by j6b performance aids. 
This past spring we conducted a tryout and demonstration of such 
training and Job Performance Aids for the AN/APN-147 and AN/ASN-35 
at Altus AFB, Okla. We took eight high electronic aptitude subjects^ 
and eight medium subjects directly from basic training at Lackland AFB. 
These subjects had received no previous technical training. These 
subjects were given a four weeks training program on how to use test 
equipment, handtools, and JPAs. These subjects were able to perform 
most of the typical organizational and intermediate maintenance tasks 
given them. 

During the coming year, we plan to conduct a controlled experi- 
ment to determine the relative effectiveness of Job Performance Aids, 
SIMMs Aids, and Technical Orders when used by personnel of varying 
levels of training} aptitude , and experiences.. We will have airmen 
who have just graduated from the appropriate Air Force electronic 
technical course, as well as, experienced personnel, perform typical 

8 
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maintenance tasks using Technical Orders » SIMMs t ype aid^ and Job 
Performance Aids > These will all be high aptitude people. We plan 
to compare their performance with novice personnel of high and average 
aptitude with a shorter amount of training but of the job oriented 
; type. These novices will use only Fully Proceduralized Job Performance 
Aids. 

24 ' THE MEASUREMENT OF. PERFORMANCE OF JOB TASKS 

Measurement of the technicians* ability to perform maintenance 
tasks as mentioned earlier has been another area of work which is basic 
to any research in, either training or technical data. The need for 
better measurement procedures was early identified as one of the 
I problems requiring research and development in connection with the * 
design of better performance aids (14). But all three services have 
addressed this problem from time to time. Some examples of such 
work are mentioned. In, the 1950 's the Air Force Personnel and 
Training Research Center as well as the Off ice of Naval Research 
supported research and development on Job Performance Measures. 
HumRRO has made use of Job Performance Measures on an ad hoc basis 
for many of their projects. In the late 1940 's and early 1950 's 
Air Training Command operated check rooms in their schools where 
students were measured concerning their ability to perform maintenance 
tasks independent of their instructors. In the early '60's U.S. Navy 
Personnel Research Activity at San Diego made extremely good use of 
such tests in their study of the proficiency of Navy electronic 
technicians in their Use of Test Equipment. But little had been 
done since the.l950's for improving job performance test technology 
for maintenance tasks per se . 

As early as 1963, some preliminary work was accomplished by our 
Training Research Division in^house. But in 1966 work began in 
earnest. An extensive review of the literature was completed in 1967 

25 (12). This work was organized around a structure similar to that 
shown before. One of the outcomes of this work was a conclusion that 
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most of the earlier work on job. performance testing had attempted to 
express the results of job performance testing under the umbrella of 
one score. It was hypothesized that tests should be developed for 
each type of job activity and that an appropriate scoring scheme should 
be. developed for each. The result would be a profile of scores for 
each subject. 

Based on these hypotheses, a contract was let to the Matrix 
Research Division of URS Systems. Job, Task Performance Tests have 
been developed,, under the direction of Dr. Edgar Schriver of that 
organization, for each type of maintenance activity for the AN/APN-147, 
doppler radar and its computer the M/ASN-35. These tests will be 
used as criterion tests in our forthcoming Job Performance Aids 
Experiment. We, also, developed some job task performance tests for 
typical organizational tasks for the UH'-IH Aircraft for Vietnamization. 
Dr. Schriver will discuss these test development efforts as well as some, 
of our more recent work on symbolic substitute tests which he has been 
doing under contract for our Laboratory. 

SUMMARY 

This morning I have given you an overview of the Air Forceps 
Research and Development Program for the improvement of Maintenance 
Efficiency, with special attention to Job Performance Aids, Job 
Orientated Training and Job Task Performance Tests. All of these 
programs have been the result of systematic study of the maintenance 
job commonly called task analysis. ' Jack Folley will follow with 
a more Indepth treatment of the Job Performance Aids Technology. He 
will be followed by Ed Shriver who will talk about measurement. We 
will then have a short panel discussion of the problems involved in 
getting the JPA technology implemented. 
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Appendix A 



Reference is made to formula presented on page 3_of the basic paper: 
M = f (E, TE, HT, D, P) 



represents Equipment 

represents Test Equipment 

represents - Hand Tools 

represents - Technical Information or 

instructions about equipment. 

represents Personnel or People assigned 

to accomplish the maintenance. 

This is a very convenient device to use when considering various 
aspects of maintenance. Its usefulness is increased when these gross 
functions are expanded into sub-functions which effect what maintenance, 
people must do or describe the attributes of maintenance people, for 
example: 

1, JE - Equipment can be expanded into 

E - The Reliability of the equipment and its components 
The more reliable the equipment the lower the amount 
of maintenance required. Some sub-systems have been 
developed to the point that they seldom fail. . 



Where ; 

E - 
TE - 
HT - 
D - 

P - 



E. - The complexity of the Equipment may be very important 
^ to the maintenance man depending on the maintenance 

concept or the type of maintenance instruction provided 
If he is required to repair or replace everything that 
can fail to the piece part level, the complexity of 
equipment is important. If he is only expected to 
identify, repair or replace a few items on an equipment 
or if he has very detailed instructions, its complexity 
may be of little importance. 

E - Design for Maintainability - If equipment is designed 
with maintenance in mind, the maintenance tasks become 
easier. 



I 
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E - The maintenance concept is closely related to maintain- 
^ ability. What should be done where? ^ If the mainte- 
nance concept calls for only "black box'* maintenance 
of the organizational or flight line level (and for 
only module maintenance at field shop or intermediate 
level, the maintenance tasks at the flight line and 
in the field shop are rather simple but those at the 
depot may be extremely difficult. 

E - Standardization of Equipment reduces the number of 
^ types of maintenance tasks to be performed. 

2. TE - Test Equipment - In a similar manner this gross function 

can be expanded into: 

TE- - General Test Equipment 
G 

TE - Special Test Equipment 
TE - Automatic Test Equipment 

3. Iff - Hand tools can be expa-nded into: 

HT^ - General hand tools 
G 

HT - Special hand tools 

4. D - Data - Technical information or instructions about 

equipment can be expanded into; 

D - Content of Data - What degree of aiding is furnished? 
^ Does the data give only information about the equipment 
organized as to equipment functions or does it give step 
by step . instructions organizational as to the tasks to 
be performed? - What is put in the book? What is the 
maintenance man expected to bring to the task job? 

D - Format of Data - size of page, size of book. 
F 

D - Mode of Presentation (Book, Audio-visual Presentation, 
^ Computer Presentation) . 

All of the functions and sub-functions affect what the maintenance man 
faces when called on to maintain a given equipment. How well he reacts 
depends on a sub-set of another gross variable the P - People or personnel 
functions. 

P - Aptitude (High, Medium or Low) 



15 



Type and length of training - were the training 
objectives general or precise? How relevant was 
the content of training to the content of the job 
or task at hand? 

P - Experience - Tile types of previous job experiences 
^ the individual has had. 

P f Motivation - How well Tnotivate6^ is he to do the task 
^ at hand? Is he rewarded for doing his job tasks 
well or is he rewarded for something else? For 
example - Ability to tell jokes. 

The P - variable also contains another set of sub-functions that 
indirectly affects the maintenance man^s behavior. 

Promotion system - is it based on his demonstrated 
ability to do the major tasks of his job? Or is it 
based on his ability to pass a paper and pencil test 
about his job or about more abstract theory? 

Organizational Structure and 

Number of Personnel - both of these may have an 
effect on his ability to be promoted among other 
things . 

Culture - How well does the man fit the traditional 
cultural norms of his job society? 

The above expansion and treatment is not intended to be complete but 
to give some idea of the thought processes involved in our Air Force Human 
Resources work concerning job analysis. An indepth consideration of some 
of the factors will indicate that some may be positive factors for mainte- 
nance. Others may be neutral, and some may be negative. 



P^ - 



N 
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APPENDIX B - SLIDES 
LIST OF SLIDES 



Slide 1 - AFHRL 

2 - Title Slide 

3 - Task Analyses of Maintenance Jobs 

4 - Picture of JPA Book 

5 - Factors Effecting Maintenance Effectiveness 

6 - A Functional Representation of the Air Force Maintenance 

Structure 

7 - AFHRL 1962 - 1963 Effort 

8 - AFHRL 1962 - 1970 Effort 

9 - AFHRL JPA Experimental Results 

10 - Same as Slide 8 

11 - PIMO Project 

12 - PIMO Experimental Results 

13 - Vietnamization 1969 - 1972 

14 - Draft Specifications and Handbooks 

15 - Same as Slide 13 

16 - JPAs for Seek Point 

17 - JPAs for Jet Engine 

18 - JPA Summary . . 

19 - Same as Slide 14 (AFHRL-TR-71-53) 

20 - Draft Specifications for Electronic Maintenance 

21 - Expanded Draft Specifications for Organizational and 

Intermediate Level 

22 - Demonstration - Altus AFB 1972 

23 - Advanced Development Experiment I 

24 - Job Task Performance Tests 

25 - Same as Slide 6 

26 - Job Task Performance Tests (AN/APN-147 & AN/ASN-35) 

27 - Job Task Performance Test (UH-IH) 

28 - Summary Slide 

29 - . AFHRL 
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Key To Abbreviations 



1. 


ASD 


Aeronautical Systems Division of AFSC 


2. 


AFHRL 


Air Force Human Resources Laboratory 


3. 


AFSC 


Air Force Systems Command . 


4. 


AMSL 


Aerospace Medical Research Laboratories 


5. 


ASA 


Applied Science Associates 


6. 


AIR 


American Institute Research 


/ • 


J>bU 


Ballistic Systems Division 


8. 


FPJPA 


Fully Proceduralized Job Performance Aids 


9. 


FPTSA 


Fully Proceduralized Troubleshooting Aids 


10. 


HumRRO 


Human Resources Research Organization 



Formerly Human Resources Research Office 
of George Washington Univeristy, 
Washington, D.C. 



11. 


JPA 


Job Performance Aids 


12. 


KENTRON 


Fully Owned Subsidy of LTV 


13. 


LRU 


Line Replaceable Unit 


14. 


LTV 


Ling-Temco-Vought , inc • 


15. 


tffS 


Maintenance Task Simulators 


16. 


PIMO 


Presentation of Information for Maintenance and Operation 


17. 


SAMSO 


Space and Missile Systems Organization 


18. 


STC 


Service Technology Corporation now KENTROJl 


19. 


XYZYX 


XYZYX Information Corp. 



